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ABSTRACT ^ 

In studies conducted in the Baltimore City Public 
Schools by the Office of Pupil and Progrdm Monitoring and Appraisal 
(OPPMA) , pre-kindergarten pupils were found to be "growing 
scOiolastically". A program. Early School Admissions Program (ESAP) 
was based on nine objectives established to prepare pre-schoolers for 
"reading readiness" when they entered the kindergarten. To assess the 
success of tlje children and the program, a criterion-measure, based 
on piagetian "concepts, was constructed in cooperation with OPPMA and 
the 'Project Manager of ESAP. Reliability and validity of the program 
Evaluation Measure (PEM) were established and found to be , 
satisfactory, thereby, giving more credence to the growth indicators 
of the pupils. The profiles of the children (th- / were tested in 
October, November, December, January and May) demonstrated a 
significant pattern of achievement similarity as reflected by Dumas 
Model for measuring profile configurations. The Dumas Model for 
determining the similarity of profile configurations enables one to 
quantify the degrae to which several profiles are shaped alike. 
(Author) 
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ABSTRACT 



In studies conducted in the Baltimore City Public Schools by the Office of Pupil and Program 
Monitoring and Apprat;al (OPPMA), pre-kindergarten pupils were found to be '^growing 
scholastically". A program. Early School Admissions Program (ESAP) was based on nine 
^objectives established to prepare pre-schoolers for "reading readiness" when they entered the 
kindergarten. To assess the success of die cliildren and the program, a criterion-measure, based 
on Piagetian concepts, was constructed in cooperation with OPPMA and the Project Manager of 
ESAP. 

Reliability and validity of the Program Evaluation Measure (PEM) were established and found 
to be satisfactory, thereby, giving more credence to the growth indicators of the pupils. The 
profiles of the children (they were tested in October, November, December, January and May) 
ctenonstrated a significant pattern of achievement sunilarity as reflected by Dumas Model for 
measuremg profile configurations. The Dumas Model for determining the similarity of profile 
configurations enables one to quantify the degree to which several profiles are shaped alike. 



Instructional Effectiveness In A Pre-Kindergarten Program: 
A Pupil Appraisal and Program Evaluation Model 
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INTRODUCTION 

One of the prime functions for the reorganization of the Baltimore City Public Schools System 
last June was to unprove the instructio lal program for all pupils. For that purpose, the Office 
of Pupil and Program Monitoring and Appraisal was established, with its primary goal being, 
to provide measurement and evaluation techniques which would be most appropriate for instruc- 
tional programs. Over the years, the evaluation of the Early Admissions Program consisted pri- 
marily of pre and post intelligence and achievement tests, practically all commercially acquired. 
This procedure, thougli widely used, has left gaps in the kind of instructional feedback that could 
be given because of one or several of the following: a) this technique often omitted the characteris- 
tics of the pupils themselves; b) sociological characteristics which would have taken into account 
variables such as one parent, two parents, relative incame or degree of educational deprivation of 
the parents when the child was first exposed to the program; c) tlie measurement of outcomes in 
terms of global standardized tests which may or may not have been related to instructional programs; 
d) strategies that apparently did not provide instructional personnel interim ideas related to individual 
or small group pupil progress, therefore, programs could not be adjusted or self-correcting as the 
school year progressed, thus enabling instructional personnel to effectively deal with each pupil. 

STATEMENT AND PROBLEM 

For all of the above reasons, Baltimore City attempted to develop a model procedure of pupil 
appraisal and program monitoring which would, in fact, correct the deficiencies of the evaluation 
strategies noted above. The problem was to develop an instruction-based measuring system which, 
in part, embraced learning constructs delineated by Jean Piaget. Intregal to the development of 
such a system is its efficiency as an assessment device that is not dependent on traditional standar- 
dized tests. In addition, such studies as one by Yawkeye (1971 ) relating to the Early School Ad- 
missions Program (ESAP) Objectives found that pre-kindergarten children, when reinforced with 
certain numerical tasks, such as consei-vation of numbers, not only were successful in developing 
rn-number concepts, but in cognitive development as well. 



STATEMENT AND PROBLEiM (continued) 



The Baltimore City study found that Early Admissions pupils improved in their ability to 
verbalize, as reflected on a criterion measure, regardless of economic backgrounds and/or 
Binet test scores. Tliis tends to uiscount findings of Audren Little in her study conducted to 
assess cognitive development in young children. 

OPPMA aspires toward a model evaluation for pre-kindergarten education which could be 
applied to other programs in grades beyond pre-kinder^arten and thus demonstrate methods 
to improve the effectiveness of school programs in general, but in particular, programs affecting 
inner-city pupils. 

STUDY HYPOTHESES 

Hypothesis One 

An intervention program such as Eariy School Admissions Program (ESAP) will impact pupils, 
as reflected on specified tests, (i.e., the Pupil Evaluation Measure), to the degree that achievement 
characteristics will be statistically non-distinguishable as a function of low income pupils coming 
from homes havmg both parents, one parent or having a guardian. 

Hypothesis Two 

The periodic administering of the criterion measure (PEM), on a monthly basis, or longer time- 
interval, will demonstrate significantly at the .05 level, a continuity of profile similarity of pupil 
increments or learning, uniquely related to each of the nine program objectives. 

PROCEDURES AND METHODOLOGY 

During October 1973, tlie total population (1,260) of the Early School Admissions Program was 
administered the instrument referred to above, the Pupil Evaluation Measure (PEM). The PEM 
had been developed by OPPMA in close collaboration and cooperation with the Project Manager 
of the Eariy School Admissions Program. 

The design for this study is characterized as a time-series testing of the total population, without 
a control group. The locally developed criterion device reflected the program objectives and re- 
quired pupils to perform tasks which, behaviorally, conformed to aspects of Piagetian constructs 
of cognitive development. 
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PROCEDURE AND METHODOLOGY (continued) 

The program objectives, (see Table 2), transcend nine different instructional disciplines, each 
showing a general degree for which cognition, as defined by Piaget, has developed in the 
individual pupil. A child's performance on the PEM criterion measure has been structured 
so that his relative performance across each cognitive dimension (i.e., concrete, semi-concrete, 
abstract and semi-abstract) can be quantified. Each dimension has contributed evenly to the 
items of the test, and each have a maximum proportionate value of twenty-five percent{25%). 

Assessment of the stability of perceptions among the monitor while administering the locally 
developed criterion measure, and some measure of the reliability and validity of the criterion 
measure (PEM) was needed. 

RESULTS 

The Early School Admissions Program (ESAP) was a pre-kindergarten program conducted for 
the purpose of developing skills that would prepare 4 year-olds for "reading readiness" unon 
their entrance into kindergarten. The pupils in the program have been identified as those 4 
year-olds living within an area defined as inner-city and disadvantaged. Family income, as 
established by the Office of Economic Opportunity, was one determinant of pupil eligibility, 
as were the limited educational opportunities of adult members of the family, dependency on 
public assistance and varied problems plaguing the family. 

Using two tests, the Stanford Binet (an intelligence quotient measure) and a locally constructed 
criterion (objective) reference measure. Pupil Evaluation Measure (PEM), OPPMA set out to observe 
each child's achievement over the school year. 

The instrument has been checked to determine its reliability through a procedure involving 
relative agreement among monitors on the PEM instrument. The inter-monitor reliability 
coefficient of the instrument was in excess of .85, thus knowledge regarding the instrument's 
administering stability was known. 

The consistency in which the pupils have performed on this test has been determined through a 
reliability computational procecfure known as the Kuder 21, Equation. 
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> RESULTS (continued) 

Interest in reliability center on how the boys and girls consistently perfonned for each month ^ 
the test was administered. As such, five measures of reliability were determined, and the ■ 
following Table reflects corresponding computational reliabilities for each of these months. 

TABLE 1 
Measures of Criterion Reliability 
for 

Five Separate Months 



MONTHS . 


RELIABILITY 


October 


0.6748215 


November 


0.5944445 


December 


0.302100 


January 


0.8117085 


May 


0.8576863 


Average 


.74 



The overall reliability average computed to .74 which reflects a moderately high reliability 
coefficient. 

Validity has been determined by comparing sub-sections of the Stanford Binef lutelligciice 
Test (Short Form) with corresponding sections of the Pupil Evaluation Measure (PEM). 
Validity is designed as the degree to which the scale measures the concept on which the 
study purports. As such, validity is the crucial test of a scale. Logical validity had been 
established primarily as a\esult of the manner in which the test has evolved. 
Previously, it has been stated that the Project Manager of the Early School Admissions 
Program was intrically involved in the development of this instrument. The items of the 
Pupil Evaluation Measure (PEM) reflect the nine program objectives of the ESAP. 

Construct validity has been assumed through a process most commonly identified as the 
Speannan Rank Order Correlation Procedure. In a publication by Haven, Rogers, and 
Lipmann, it is suggested that validity can be estimated by observing the comparability rf 
performance of one instrument to performances by the Subjects on an existing instrument. 
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RESULTS {contirj:ed) 

''Sometimes, several independent criteria are available 
to measure a single dimension, and the researcher wishes 
to devise a simpler measure to assess the dimension, so, 
he may relate the newly constructed measures to these 
criteria. " 

In assessing the Pupil Evaluation Measure scores ranked by school, and comparing them 
with the Stanford Binet rank order scores, there was found to be higli validity between 
the two instruments. Data from both instruments were collected in October when the 
Stanford Binet was administered as a pretest and the PEM was administered as a measure 
of the degree to which the program objectives were met. PEM was administered each month 
thereafter, other than February, March and April, ^ue to system-wide reasons. 

The items in PEM and the sub-tests of the Stanford Binet were observed carefully and found 
to be related. Chein defines construct validity as the degree to which phenomena can be ob- 
^ — served on separate criterion designed to assess similar traits, conditions or phenomena. The 
construct validity coefficient, so designed, using these instruments during the period between 
October and May, was moderately high at the .84 level. 

Based on previous reports in the literature and general positions espoused by many educators, 
the data was examined for six months from the first admiristration jf PEM in October 1973 
to determine if there were relative similarities of pupil performance on the instrument in terms 
of specified sociological and demographic variables. The variables examined were one parent 
in the home, two parents in the home, guardian, aid to families with dependent children and 
non-aid to families with dependent children. 

The criticalness of these variables are accentuated by the fact that insight can be more 

ft 

realistically determined as to the relative effectiveness of intervention programs such as 
ESAP are having on a more comprehensive basis. 
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RESULTS (continued) 

»• 

Using information gathered from Pupil Personal History Fomjs, each child was categorized 
according to his socio-economic background. The categories and test results, according to 
each of the nine objectives, follow: 

A. Pupils having in the home: 

1. Both Parents (B) 

2. One Parent (O) 

3. Guardian Parents (G) 

B. Receiving Aid To Dependent Children: 

1. No (N) 

2. Yes (Y) 

C. Schools with: 

1. All Black Students 

2. All White Students 

3. Mixed Students 

After many observations, evaluations and assessments, the analysis proved to be rewarding, 
"our children were growing". Some were growing faster than others and some children 
achieved higher levels than others, but they were achieving, each at his own rate. The 
results indicated each child to have weaknesses in some skill areas and strengths in others. 
This indicated the possibility of building on strengths to improve weaknesses. 

The evaluation of each objective of the program was done so as to follow the child's develop- 
ment in his ability to transfer from concrete reasoning to the level of abstract reasoning. Data 
were included to den»onstrate achievement by child, by class, by school, by program and by 
pupil background. 
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RESULTS (continued) ' 

The purpose of the evaluation by objective was to be able^to determine the levels of 
development dem.onstrated by pupils. A determination could be made as to areas where 
a pupil maymeed heavy concentration in order to help hini develop in the acquisition of 
skills. These skills included classification, verbal, visual and auditory perceptual, motor 
perception, recognition and use of art media and the ability to reason and think through 
problems. Another ''must area" was self-concept, children need to "feel good" about them- 
ieUes to be successful. The scoring was as follows: 

0 - 24 - demonstrated the pupil working at concrete level 
' 25 - 49 - demonstrated ihe pupil at the semi-concrete level 

50 - 74 • dei^onstrateci the pupil at the semi-abstract level 
75 -.100 - demonstrated the pupil achieving at the abstract level 

^ The following data indicate the number of children in the Early School Admissions Program 
who were below or above the semi-concrete/semi-abstract stage of cognitive development in 
October 1973 and how they appeared in June of 1974. Tlie summaries indicate, by objective, 
those above the 50% levels. 
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ILLUSTRATION OF ACHIEVEMENT PATTERNS 
FOR FIVE SEPARATE MONTHS FOR PUPILS 
COMING FROM HOMES OF 2P, IP, G, ADC, NON-ADC 

TABLE 2 
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-OCT NOV DEC JAN FEB MAR APR MAY OBJECtlVE I 

Each pupil will demonstrate the ability to identify, name and describe a variety of concrete 
objects including people, animals and toys. 



Time Tested 
October 
June 



Below 50% 
755 
230 



Above 50% 
262 
845 



-90 
-75 
.60 
-45 
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-30. 

























— \ 



















































OCT NOV DEC JAN FEB MAR ^PR MAY OBJECTIVE 2 

Each child will demonstrate individual development of furthering his language ability in 
conversation, self-expression and relating experiences. 



Time Tested 
■ October 
June 



Below 50% 
632 
185 



Above 50% 
294 
823 
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— :— ONE PARENT 

BOTH PARENTS 

— GUARDIAN 
r- NON-ADC 
AOC 



-50 
-75 

I 

\ 
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ILLUSTRATION OF ACHIEVEMENT PATTERNS 
)R FIVE SEPARATE MONTHS FOR PUPILS 
COMrNG FROM HOMES OF 2P, IP, G, ADC, NON-ADC 

TABLE 2 
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-OCT NOV DEC JAN FEB MAR APR MAY OBJECTIVES 

Each child will use problem situations to develop thinking and reasoning skills. 



Time Tested 
October 
June 



Belovy ^0% 
452 \ 
60 



Above 50% 
664 
.1063 



-90 
-75 
-60 
-45 

I 
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--30. 
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OCT NOV, DEC JAN FL3 MAR APR MAY OBJECTIVE 4 

Each pupU will develop classification skills and will be able to group concrete objects by 
form, size, color and function. 



Time Tested 
October . 

June 



Below 50% 
933 
126 



Above 50% 
357 , 
6S5 
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BOTH PARENTS 
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ILLUSTRATION OF ACHIEVEMENT- PATTERNS 
FOR FIVE SEPARATE MONTHS FOR PUPILS 
COMING FROM HOMES OF 2P, IP, G, ADC, NON-ADC 

TABLE 2 
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.-OCT • NOV DEC JAN FEB MAR APR MAY OBJECTIVE 5 

Each pupil will develop skills in visual and auditory perception and will be able to 
discriminate, memorize, recall and. reproduce. 

Below 50% • Abov 50% 

297 



Time Tested 
October 
June 



927 
141 



687 



-90 
-75 
-60 

-45 
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-30 
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OCT NOV DEC JAN FEB MAR APR MAY . OBJECTIVE 6 

Each pupil will develop motor perceptual skills and will be able to identify, name and 
describe body parts and functions, orient one's body to different kinds of spaces, and 
coordiante eye-hand and eye-foot actions. 

Time Tested • Below 50% Above 50% 

October 897 .524 

June '218 537 ' 
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ILLUSTRATION OF ACHIEVEMENT PATTERNS 
FOR FIVE SEPARATE MONTHS FOR PUPILS 
COMING FROM HOMES OF 2P, IP, G, ADC, NON-ADC 

TABLE 2 



-OCT NOV DEC JAN FEB MAR APR MAY OBJECTIVE 7 

Each pupil will develop quantitive skills by bf '^, able to identify, match and compare 
J* sets of objects, count, and develop basic understanding in addition and subtraction, 
as reflected. 

Time Tested Below 50% Above 50% 

October 798 224 . 

June • 301 ' 772" 



-SO 

-60 
-45 

_30. 
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OCP NOV DEC JAN FEB' MAR APR MAY OBJECTIVES 

Each pupil will develop wholesome feelings of self as reflected by his ability. 

Time Tested , Below 50% Above 50% 

October « 934 341 

June 174 776 



ONE PARENT 
BOTH PARENTS 



FR?rKEY — GUARDIAN . '^^'^ 
. NON-ADC 



12. 
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-OCT NOV DEC JAN FEB MAR APR MAY OBJECTIVES 

Each pupil will have opportunities to increase those abilities that allow him to use a 
variety of art media and many art forms. 

Time Tested Below 50% AboVe 50% 

October 1072 661 

June 2 . 345 



The data demonstrated the reading readiness of boys and girls in the Early School Admissions 
Program as they entered kindergarten in September of 1974. More than 77% of the pupils 
appeared ready for the kindergarten program and beyond. 

Included within this report are additional charts explaining other aspects of the studies 
(racial, social, economic). The in-depth analysis included herein proves the validity and 
reliability, as well as the value of this report. 
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MEASURES OF PROFILE SIMILARITIES OF PUPIL ACHIEVEMENT 
AS OBSERVED OVER A FIVE-MONTH-TLME SPAN 

Profile analysis, such as' defined by Dumas, is the study of profile sLmibrity between test 
scores, and it has a position of\eal value in the research field. Probably no phase of educational 
administration has received more^^ticism than the evaluation of performance. Profile analysis 
and the application of one of /the techniques currently available will permit far better evaluation 
of efficiency than any of tiie old rating scale techniques. 

The sfanilarity of continuity of growth, as reflected on the criterion measure, indicated a 
pattern of similarity tiiat was significani beyond the .05 level. The Dumas Model of Profile 
Sfanilarity computed to a value of J45 overall for the school year ending June 1974. Such 
analyses would tend to reflect program impact that has generally resulted in pupils, affected 
by this program, demonstrating differential achievement trends across the nine program objec- 
tives, but more important, demonstrating continuity of achievement during the school yenr. 
Table 3 pictures even better the profile similarities of achievement across the five months of 
testing. 
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DISCUSSION 



The importance of the problem is accentuated by the fact that of the numerous in-house 
and cutside evaluators variously involved in former evaluations, the product emphasis 
apparently has been more in terms of program implementation, the operational aspects, 
and subjective reactions on the part of evaluators basing and concluding programmatic 
impact as a function of anecdotal references. Typical of the manner m which evaluations 
of the Early School Admissions Program has been affected or implemented, one finds in 
the official report of the 1966-67 ESEA Early School Admissions Program, on page 4, one 
notes: 

"It is difficult, on the basis of a short observation, to note tangible 
evidence of possible effect of a program on the children involved. 
There were, however, some differences in varying degrees, noted by 
the three evaluators. During each brief school visit, the team members 
divided their efforts as follows: One member of the team interviewed 
parents in a group situation; while another, when possible, talked with 
children now in kindergarten and with Non-Early Admission children 
vho were in the program last year; and the third member observed and 
talked with children and interviewed parents at each center visit. 

Findings were then later compared school-by-school with the evaluation 
compared by each center at the close of the school year. However, the 
reported findings are not related to the individual schools. The material 
was written after discussing procedures. They were then collated, typed, 
and reacted to by each individual team member, prior to preparing the 
final report. The evaluators feel that the final report is a team effort 
and represents their best combined judgment within the limitation imposed. 

The validity of the report thus rests on the impressions received and reacted 
to on the basis of the experience and philosophy of the evaluators, as well 
as on the accuracy of the information provided by the school personnel/' 
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DISCUSSION (continued) 

Serious concerns are in order if the preceding quote typifies evaluational procedures previously 
used on the ESAP. Another rather extensive evaluation submitted by a former Research 
Director, one finds extensive information on demographic, social, economic and medical type 
information being presented, but programmatic impact on the part of the pupils is never 
specifically delineated or attempted to be delineated other than generalized intuitive reactions 
on the part of the evaluator. 

Obviously, insight is needed relative to whether or not the program itself is affecting the 
boys and girls and this should be, as thoroughly as possible, emperically substantiated. This 
has been the underlying basis of the current report which essentially attempts to delineate 
a multi-variant approach in terms of how girls and boys are being affected and impacted by 
the Eariy School Admissions Program operating during the 1973-74 school year in Baltimore 
City. Nine specific objectiv^haye been delineated, and the relative impact and inter-relation- 
ship of these different objectives. has also been observed and documented. 

In addition, documentation of the program impact on pupils has been coupled with a construct 
which relates to a Piagetian definition of cognitive development, on several dimensions of cogni- 
tive growth on the part of pupils. Efforts have been made to make associations as far as these 
nine objectives are concerned to specified aspects of the Piagetian concept of cognitve develop- 
ment. Some intriguing issues, as addressed by this study, were determining 1) whether pupils 
are t^ing affected by an injervention program such as the Early School Admissions. Program; 
2) to ^'^termine the various ways or dimensions in which such effect is actually taking place 
with the children; 3) to determine whether or not specified sociological differences can be 
determined as a function of boys and girls being exposed to this program; 4) to determine 
the longitudinal implications as a function of observing boys and girls, who were former Early 
School Admissions pupils, who may now be in the kindergaiten or other grades, and determine 
if their subsequent achievement pattern indicates advantage over boys and girls who may not 
have had either an Early School Admissions Program experience or any pre-kindergarten ex- 
perience; 5) the manner in which mechanisms con be set up sensitive enough to determine 
to what extent validation procedures can be used for varifying the fact of observed discrepancies 
of different observations. 
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DISCUSSION (continued) 

It is extremely important that educators acquire the insight and sensitivity as public school 
systems increasingly progress toward individualization, and maximally adapt school systems 
to accommodate the unique needs of individual pupils. Public school officials should be 
cognizant of the extent in which instructional experiences are broad and comprehensive and 
can be delineated according to some pedogogic philosophy or educational learning construct- 
Program impact and school experiences can be observed more practically in a manner which 
is most relevant to the individual child. The approach for assessing the effects of this program 
on the various boys and girls in the project has been established to the degree that we now 
see or have seen how boys and girls achieve, and while their relative growth rates rnay not be 
the exact level across all objectives, achievement has been observed across all objectives. 

One of the major ramifications of this study relates to the manner in which the individual 
pupfl may be observed in a comprehensive manner which allows continuous observation on 
that individual over several variables, factors, or objectives. A possible use on other grade 
levels of the school system are rather apparent as far as a similar model can be devised that 
allows the individual pupii to be observed or assessed across several related factors in one 
particular skill area. 

The approach for assessing the effects of this program on the various boys and girls in the 
project has been established to the degree that we now see or have seen how boys and girls 
achieve, and while their relative growth rates may not be the exact levels across all objectives, 
achievement has been observed across all objectives. 

SUMMARY 

The Office of Pupil and Program Monitoring and Appraisal (OPPMA.) has found a means 
of evaluation that may have far-reachmg effects on the educational system of the Bahhnore 
City Pubjic Schools. Discovering learning patterns at the pre-kindergarten level can enable 
educators to prepare programs that may build a more effective educational process within 
the system. More children can be successful when the rate of learning is recognized early. 
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